Head-mounted image display device and data processing apparatus including the same

ABSTRACT

A head-mounted image display device with a good operability and a data processing apparatus including such a display device. A head-mounted image display device wherein a liquid crystal panel and an enlarging optical system are crystal display panel through the enlarging optical system housed in a frame and wherein images generated on the liquid are visible in the field of view in front of the frame comprises a first direction detection means ( 107 ) disposed in part of the frame ( 101 ) for detecting the direction of the user&#39;s head; an observation point detection means ( 109 ) disposed in part of the frame for detecting the user&#39;s line of sight; a trigger input means ( 4 ) for transmitting trigger signals; and an image generation means ( 3, 105 ) for controlling images generated on the liquid crystal display panel based on signals from the first direction detection, observation point detection, and trigger input means.

This application is a divisional of application Ser. No. 09/330,397filed on Jun. 11, 1999 now U.S. Pat. No. 6,320,559, which is adivisional of application Ser. No. 08/582,997 filed Feb. 12, 1996, nowU.S. Pat. No. 5,977,935 which is International ApplicationPCT/JP94/01340 filed on Aug. 12, 1994 and which designated the U.S., andwas not published in English, claims the benefit thereof andincorporates the same by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a head-mounted image display device fordisplaying images on a screen showing a virtual space and to a dataprocessing apparatus including this device.

2. Description of the Prior Art

The design of electronic circuits or experiment thereon requiresmaterials such as data sheets due to the need to confirm thespecifications of parts to be used. In this case, the materials arescattered on and around the designer's desk, thereby limiting theworking space or hindering the designer from finding required materialsquickly. This reduces the working efficiency.

To carry out design efficiently, computer aided engineering (CAE) forworking out designs using computers has been introduced. That is, apaperless design environment has been established in which, for example,materials such as data sheets are registered in a computer to enablethem to be displayed and referenced on the screen of the display asrequired. The above inconveniences have thus partly been eliminated.

If, however, a large number of parts are to be used and a large numberof materials must be referenced at the same time, although severalwindows can be opened on the screen to display different materials inthe different windows, the windows must sometimes overlap each other ifa large number of windows must be used simultaneously due to thephysically limited size and resolution of the CRT display for displayinginformation. As a result, the amount of information that can bereferenced at the same time is limited, and the efficiency provided bythe introduction of CAE is thus limited; in some cases, it is faster tofind required materials among those scattered on and around the table asdescribed above.

The working efficiency is also reduced in the maintenance or repair ofan installed apparatus, wherein the operator must carry out such anoperation while referencing instruction manuals and wherein because thesurroundings are in disorder, the operator cannot place the manuals inpositions in which they can be referenced appropriately. Under suchconditions, however, it is further difficult to place a CRT displayclose to the operator, and the introduction of CAE is impossible. Thus,no other appropriate solutions were available.

Several methods of mounting a display apparatus on the operator's headand displaying information from materials thereon have thus beenproposed.

Published Unexamined Patent Application No. 5-100192 describes a methodof displaying screens from electronics such as word processors on aspectacle-like display, but this is only a spectacle-shaped variation ofconventional displays and is not suited to the presentation of a largenumber of materials. Published Unexamined Patent Application No.4-263283 proposes a portable virtual reality apparatus, but this simplyprovides an artificial reality apparatus and is not suited to themanipulation of displayed data.

In addition, for conventional head-mounted image display devices,several problems arising from the need to alternatively view a virtualimage on the liquid crystal display and the corresponding image on theexternal keyboard due to the large difference in image quality betweenthese images, have been pointed out. To solve this problem, someconventional techniques cause a display image and the correspondingexternal image to overlap each other. Head-mounted image display devicesof this type, however, also have problems. For example, in suchapparatuses, the visibility is very bad because external images aresimply caused to overlap the corresponding display images, andalternatively viewing different images may strain the operator's eyes.

In addition, in head-mounted image display devices, when, four example,data is input to a word processor, keys to be pressed must be confirmedwhile viewing a display image, so the external keyboard must be checkedwhile watching the display image.

To solve this problem, a means for switching between an external imageand the corresponding image from an arithmetic operational device may beprovided, or a configuration that enables the use of part of the fieldof view for watching the display while using the other part for checkingthe exterior may be provided, as proposed in Published Unexamined PatentApplication No. 5-100192. In this case, however, since data must beinput while using a particular part of the field of view to check theinput device, mounting the image display device on the operator's headeliminates the freedom of the operator's inputting position, and longtime work thus results in a significant fatigue. In addition, if akeyboard is used as an input device, the input operation is verydifficult when the keys are larger than the tips of the operator'sfingers, so the limit of the practical size of portable terminalsdepends on the size of the keyboard. Furthermore, due to the fixedarrangement of keys in conventional keyboards, optimal key arrangementsfor various applications are not available. In this information-orientedsociety, more and more attention is being paid to multimediaapplications, so people must often deal with data in which motion andstill pictures are mixed. Under these circumstances, the need ofcomplicated procedures for creating motion pictures hinders peoplewithout expertise from providing adequate expressions using multiplemedia effectively. For hypermedia applications, HyperCard commerciallyavailable from Apple Co., Ltd is popular, but requires a large number ofmouse clicks, thereby possibly resulting in mistakes in clickingpositions.

For movie postproduction processes for processing, synthesizing, andediting movie materials, a large amount of time is required to completethe process due to the need to accurately carry out complex proceduresaccording to pre-calculations, and the flexibility of these processes islimited; it is impossible to test various combinations of images duringthe process.

In addition, if a head-mounted image display device is used, only theperson wearing the device can view image display information output froma data processing apparatus. If a plurality of people have to view imagedisplay information output from the data processing apparatus, then:

-   (1) these people must use a single head-mounted image display device    in turn, or-   (2) another data processing apparatus with a CRT display must be    provided.

In this manner, a plurality of devices may be required according to aparticular purpose and usage.

Attempts are being made to reduce the size of computers and wordprocessors, and the utilization of portable data processing apparatusesis increasing. Such portable data processing apparatuses ordinarilyinclude an open type display that is opened in use. However, with thefurther reduction of the size of portable data processing apparatuses,the extension of the range within which such devices can be used, andthe increase in demand for functions of publicly creating confidentialdocuments, the use of head-mounted image display devices has beenproposed. As in Published Unexamined Patent Application No. 5-100192,such apparatuses comprise an arithmetic processing device and ahead-mounted image display device to display screen information outputfrom the arithmetic operation device, through the head-mounted imagedisplay device as enlarged virtual images. In such conventionalstructures, the arithmetic operation device and the head-mounted imagedisplay device are always separated from each other, they must becarried separately, resulting in poor portability despite their reducedsizes. Even if the head-mounted image display device is carried in anouter sheath case, handling becomes cumbersome, and faults are likely tooccur.

BRIEF SUMMARY OF THE INVENTION

It is an objective of this invention to provide a head-mounted imagedisplay device suitable to efficient design, experiment, maintenance,and repairs.

It is another objective of this invention to provide a head-mountedimage display device that has good portability and visibility and thatdoes not affect the user's health.

It is yet another objective of this invention to provide a head-mountedimage display device that enables the operator to input data in anatural position without depending on the size of an input device.

It is yet another objective of this invention to provide a head-mountedimage display device that can be used for multimedia applications andthat enables the user to check both an image display section anddisplayed information on the attributes of a displayed image simply bychanging the direction of the field of view when editing still andmotion pictures.

It is still another objective of this invention to provide ahead-mounted image display device that uses as an image retrieval methodthat is complicated when used with hypermedia applications a method fordisplaying retrieved attributes based on time series or hierarchies andthat reduces mistakes attributable to the image retrieval method simplyby switching the direction of the field of view between a first displayarea and a second display area.

It is still another objective of this invention to provide ahead-mounted image display device that enables the three-dimensionaldisplay of information displayed in the second display area the amountof which varies with changes in space or time by changing the parallaxof the operator's eyes, thereby reducing mistakes in inputtinghierarchical data.

It is still another objective of this invention to provide ahead-mounted image display device that can express even minor positionalrelationships in an image during edition in imaginative movie productionmethods in which voices are supplementarily input to process and editmovie materials and that in the edition of motion pictures, can expressconditions based on time series such as partial acceleration anddeceleration in such a way that they look like real conditions.

It is still another objective of this invention to provide a dataprocessing apparatus that can simply switch a display means between ahead-mounted image display device and a screen or the like as requiredto display thereon image display information output from the dataprocessing apparatus main body.

It is still another objective of this invention to provide a dataprocessing apparatus comprising a head-mounted image display device thatis portable and easy to handle.

(A) A head-mounted image display device according to one aspect of thisinvention wherein a liquid crystal panel and an enlarging optical systemare housed in a frame and wherein images generated on the liquid crystaldisplay panel through the enlarging optical system are visible in thefield of view in front of the frame, comprises a first directiondetection means disposed in part of the frame for detecting thedirection of the user's head, an observation point detection meansdisposed in part of the frame for detecting the user's line of sight, atrigger input means for transmitting trigger signals, and an imagegeneration means for controlling images generated on the liquid crystaldisplay panel based on signals from the first direction detection,observation point detection, and trigger input means.

According to another aspect of this invention, the above head-mountedimage display device has a second direction detection means mounted onthe user for detecting the direction of the user's body to transmit thedetection signal to the image generation means.

According to yet another aspect of this invention, in the abovehead-mounted image display device, the image generation means drives theliquid crystal display panel to display an image on a virtual screen asa virtual image, selects and identifies an arbitrary image from theplurality of displayed images based on the output of the observationpoint and direction detection means, and specifies the identified imageas those to be manipulated on the basis of the output of the triggerinput means.

According to still another aspect of this invention, in the abovehead-mounted image display device, the image generation means moves thearbitrary selected image to the positions on the virtual screen whichare determined on the basis of the output of the observation point anddirection detection means and stops the movement in response to theoutput of the trigger input means.

With the above configuration, in designing circuits or experimentingthereon, this invention can display on a virtual screen fixed to thespace those materials such as data sheets which are used to confirm thespecifications of parts to be used, thereby preventing the working spacefrom being limited by a large number of materials to avoid the reductionof efficiency even when a large number of parts must be used.

In carrying out the maintenance or repair of an installed apparatus,since the locational relationship with the operator can be displayed ona fixed virtual screen, efficient operations are ensured even if thereis no appropriate place in which manuals can be put.

Furthermore, due to the ability of this invention to freely movearbitrary virtual screens, required screens can be moved to positions inwhich images can be seen most clearly, thereby eliminating the need forthe operator to assume an unnatural posture.

The movement of virtual screens does not require the use of theoperator's hands, so they can be solely used for operations.

(B) A head-mounted image display device according to another aspect ofthis invention wherein a liquid crystal panel and an enlarging opticalsystem are housed in a frame and wherein images generated on the liquidcrystal display panel through the enlarging optical system are visiblein the field of view in front of the frame, has a screen disposed in thedirection in which enlarged virtual images generated by the enlargingoptical system are formed.

According to another aspect of this invention, the above head-mountedimage display device further has a synthesizing optical means forsynthesizing an image showing the exterior of the user and an image onthe liquid crystal display panel.

According to yet another aspect of this invention, the abovehead-mounted image display device employs part of an outer sheath casefor a portable computer as a screen.

With the above constitution, this invention implements an image displaydevice that has good portability and visibility and that does not affectthe operator's health by providing an optical system for synthesizing animage on the liquid crystal display and an image of the exterior andalso providing a screen in the direction in which images on the liquidcrystal display are formed, or by using a single eye-type head-mountedimage liquid crystal display device with a screen disposed in thedirection in which vertual images on the liquid crystal display areformed. A portable computer that has good portability and visibility andthat does not affect the operator's health can also be implemented byemploying part of an outer sheath cover of the computer as a screen.

(C) A head-mounted image display device according to yet another aspectof this invention wherein a liquid crystal panel and an enlargingoptical system are housed in a frame and wherein images generated on theliquid crystal display panel through the enlarging optical system arevisible in the field of view in front of the frame, has an input devicewith a sensor for inputting signals and an image generation means forassigning input keys to locational information from the input device andsynthesizing on/off information on the input keys and the results ofarithmetic operations to allow the liquid crystal display panel togenerate synthesized images.

According to yet another aspect of this invention, the abovehead-mounted image display device displays input information on part ofthe display screen and the results of arithmetic operations on the otherpart.

With the above configuration, this invention enables the user to inputdata in a natural position without viewing the input device, and tocheck the conditions of the input device without substantially movingthe user's field of view. In addition, since a key assignment means inthe arithmetic operation device executes key assignment, an optimalkeyboard for a particular application can be provided regardless of thenumber of keys in the input device, and various interface screens can bedisplayed in a virtual keyboard display section. Furthermore, the inputdevice does not require the size and arrangement of input keys to bedetermined in advance, so the use of a small tablet enables a smallportable terminal to be realized.

(D) A head-mounted image display device according to still anotheraspect of this invention wherein a liquid crystal panel and an enlargingoptical system are housed in a frame and wherein images generated on theliquid crystal display panel through the enlarging optical system arevisible in the field of view in front of the frame, has a locationdetection means for detecting the field of view in the horizontal orvertical directions, a voice input means joined with the frame using aflexible joint, and an image generation means for controlling imagesgenerated on the liquid crystal display panel based on the locationdetection and voice input means.

According to still another aspect of this invention, in the abovehead-mounted image display device, the voice input means is composed ofa small microphone that enables unspecified speakers to input voicesthereto.

In addition, a head-mounted image display device according to stillanother aspect of this invention wherein a liquid crystal panel and anenlarging optical system are housed in a frame and wherein imagesgenerated on the liquid crystal display panel through the enlargingoptical system are visible in the field of view in front of the frame,has a location detection means for detecting the field of view in thehorizontal or vertical directions and an image generation means forsegmenting the display area into a first display area for displayingimages and a second display area for displaying attributes of an imageand switching between the first and second display areas based on theoutput of the location detection means.

According to still another aspect of this invention, in the abovehead-mounted image display device, the image generation means can switchbetween the first and second display areas based on the output of thelocation detection means, and controls attribute data shown in thesecond display area according to voice input signals.

In addition, a head-mounted image display device according to stillanother aspect of this invention wherein a liquid crystal display paneland an enlarging optical system are housed in a frame and wherein imagesgenerated on the liquid crystal display panel through the enlargingoptical system are visible in the field of view in front of the frame,has a location detection means for detecting the field of view in thehorizontal or vertical directions and an image generation means forsegmenting the display area into a first display area for displayingimages and a second display area for displaying attributes of an imageand switching between the first and second display areas based on theoutput of the location detection means to display in the second displayarea for displaying attributes of an image a geometry or operation modelto be incorporated in the first display area.

According to still another aspect of this invention, in the abovehead-mounted image display device, geometrical data or camera work to beincorporated in the first display area is displayed in the seconddisplay area.

According to still another aspect of this invention, in the abovehead-mounted image display device, space-time area data to beincorporated in the first display area is displayed in the seconddisplay area.

According to still another aspect of this invention, in the abovehead-mounted image display device, space-time area data to beincorporated in the first display area is three-dimensionally displayedin the second display area with a parallax provided therein.

In addition, a data processing apparatus according to still anotheraspect of this invention comprises a head-mounted image display devicewherein a liquid crystal display panel and an enlarging optical systemare housed in a frame and wherein images generated on the liquid crystaldisplay panel through the enlarging optical system are visible in thefield of view in front of the frame; and a main body including a centralprocessing unit and a keyboard, with the image display device and themain body connected together via a connector.

According to still another aspect of this invention, in the above dataprocessing apparatus, the main body has a socket in which the liquidcrystal display panel is mounted, the liquid crystal display panelmounted in the socket and the liquid crystal display panel provided inthe head-mounted image display device are each detachably constituted,and the liquid crystal display panels are also configured so as to beshared by the main body and the head-mounted image display device.

With the above constitution, this invention enables the operator toprecisely adjust the sizes of images and relative relationships amongthem during edition while viewing the images, by providing the locationdetection means for detecting the field of view in the horizontal orvertical directions and the voice input means joined with the frameusing a flexible joint.

Since the voice input means comprises a small microphone that enablesunspecified speakers to input voices thereto, any user can wear thismeans. Furthermore, by inputting a form of voice input processing usingWordSpotting on a neuro-computer, mistakes in input procedures aresomewhat compensated for because associated keywords can be recognizedon the basis of certain probabilistic determination, thereby enablingedition to be executed without the need to memorize all the commands.

Since the location detection means can switch between the first andsecond display areas, the display area can be spatially classified intoa plurality of pages, thereby enabling the overall picture to beconfirmed during edition, in contrast to conventional techniques whereinan image and its attributes are displayed in a single page constitutinga display area.

Since the second display area for displaying attributes of an imagedisplays a geometry or operation model to be incorporated in the firstdisplay area, it can display not only image materials required duringedition but also materials used during interactive input, and symbolsdisplayed in the second display area can be input by clicking on themwith a mouse, thereby reducing operational mistakes.

Since the second display area for displaying attributes of an imagedisplays space-time area data to be incorporated in the first displayarea, information that varies with the lapse of time can be processedappropriately.

Since the second display area for displaying attributes of an imagethree-dimensionally displays hierarchical and space-time area data to beincorporated in the first display area with a parallax provided therein,the depth of images in the first display area and spatial-temporal areadata can be edited and processed appropriately.

Although the above display method has been described in conjunction withan embodiment that mainly displays images and in which an image isdisplayed to the right of its attributes, this invention is not limitedto this arrangement and attribute data may be located to the left of theimage or over or under the image.

(E) A data processing apparatus according to still another aspect ofthis invention comprises a head-mounted image display device wherein aliquid crystal display panel and an enlarging optical system are housedin a frame and wherein images generated on the liquid crystal displaypanel through the enlarging optical system are visible in the field ofview in front of the frame; and a main body including a centralprocessing unit and a keyboard, with the image display device and themain body connected together via a connector.

According to still another aspect of this invention, in the above dataprocessing apparatus, the connector has a power supply for driving theimage display device and an information transmission means fortransmitting image display information to the image display device.

According to still another aspect of this invention, in the above dataprocessing apparatus, the main body has a socket in which the liquidcrystal display panel is mounted, the liquid crystal display panelmounted in the socket and the liquid crystal display panel provided inthe head-mounted image display device are each detachably constituted,and the liquid crystal display panels are also configured so as to beshared by the main body and the head-mounted image display device.

With the above configuration, this invention enables the display meansto be switched simply and quickly as required. Images displayed on thehead-mounted image display or screen or the like are larger than thoseon conventional displays, so the screen in this invention has a largecapacity. Furthermore, since a single device can accommodate a pluralityof usages, it is not necessary to use different devices for differentpurposes.

Conventional apparatuses require two cords for supplying power to theimage display device and for transmitting image display information tothe same device because the main body and the image display device eachhold a power supply. In this invention, however, the connector includesthe power supply for driving the image display device and theinformation transmission means for transmitting image displayinformation to the image display device, as described above, so a singlecord can be used to supply power to the image display device as well asto transmit image display information to the same device.

(F) In a data processing apparatus according to still another aspect ofthis invention comprising a portable arithmetic operation processingdevice and a head-mounted image display device for displaying imageinformation from the arithmetic operation processing device, thearithmetic operation device has a housing section for housing the imagedisplay device while unused and the image display device comprises aliquid crystal display panel provided so as to correspond to at leastone of the user's eyes, an enlarging optical means for forming enlargedvirtual images on the liquid crystal display, and a holding means forholding the liquid crystal display and the enlarging optical means infront of the user's face and fixing the entire apparatus to the user'shead.

According to still another aspect of this invention, in the above dataprocessing apparatus, the arithmetic operation processing device has acharger having an electric contact in its housing section, and the imagedisplay device has an electric contact formed so as to connect to theelectric contact in the housing section when the image display device ishoused in the housing section and a battery charged when both contactsare connected together.

With the above constitution, due to its integral structure in whichimage information from the portable arithmetic operation processingdevice is displayed on the head-mounted image display device, which ishoused in the arithmetic optation processing device, this inventionenables confidential documents to be prepared, eliminates a locationallimitation in which the apparatus cannot be used unless the display isfully opened, and improves the portability and operability of the dataprocessing apparatus. In addition, since the battery for driving thehead-mounted image display device can be charged immediately afterhousing, and the arithmetic operation processing device and thehead-mounted image display device communicate with each other by radio,the arithmetic operation processing device and the head-mounted imagedisplay device are completely separated from each other, therebyallowing the data processing apparatus to be handled easily in its useand significantly simplifying the removal and mounting of thehead-mounted image display device in the housing section.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a head-mounted image display device according to oneembodiment of this invention;

FIG. 2 is a block diagram showing the details of the display device inFIG. 1;

FIG. 3 shows a scene as seen by the operator in the embodiment in FIG.1;

FIG. 4 is a block diagram showing the details of a control system inFIG. 2;

FIG. 5 is a flowchart showing the operation of the control system inFIG. 4;

FIG. 6 is a flowchart showing a method for specifying images in theembodiment in FIG. 1;

FIG. 7 is a flowchart showing a method for moving images in theembodiment in FIG. 1;

FIG. 8 shows a head-mounted image display device according to anotherembodiment of this invention;

FIG. 9 shows a scene as seen by the operator in the embodiment in FIG.8;

FIG. 10 shows a head-mounted image display device according to yetanother embodiment of this invention;

FIG. 11 describes a screen applied to the embodiment in FIG. 10;

FIG. 12 explains a usage of the embodiment in FIG. 10;

FIG. 13 describes a usage of a head-mounted image display deviceaccording to still another embodiment of this invention as seenlaterally;

FIG. 14 explains the usage of the embodiment in FIG. 13 as seenfrontways;

FIG. 15 describes the operator inputting data with a head-mounted imagedisplay device according to still another embodiment of this invention(which displays input information and the results of arithmeticoperations) mounted on the operator's head;

FIG. 16 explains the display screen of the image display device in FIG.15;

FIG. 17 is a block diagram showing the constitution of an arithmeticoperation device in the embodiment in FIG. 15;

FIG. 18 is a flowchart showing the flow of signals in the embodiment inFIG. 15;

FIG. 19 is a flowchart showing a process executed in an assignment meansin FIG. 17;

FIG. 20 shows the configuration of the image display device in theembodiment in FIG. 15;

FIG. 21 is a perspective view showing a data processing apparatus,according to one embodiment of this invention, which is used for apersonal computer and a spectacle-like display used as a display meansthat is an example of the head-mounted image display device;

FIG. 22 is a side view showing the details of a head-mounted imagedisplay device for the data processing apparatus in FIG. 21;

FIG. 23 is a perspective view showing a conventional data processingapparatus in which a main body and a display section can be separatedfrom each other;

FIGS. 24 and 25 show the relationship between the vertical movement of aliquid crystal display panel and the position of an image;

FIGS. 26 and 27 show the relationship between the longitudinal movementof a liquid crystal display panel and the size of an image;

FIG. 28 is a perspective view showing a support mechanism for the liquidcrystal display panel;

FIG. 29 is an exploded perspective view of FIG. 28;

FIG. 30 is a perspective view showing a conventional data processingapparatus in which a main body and a display section can be separatedfrom each other;

FIG. 31 is a perspective view showing a data processing apparatus towhich an image display device according to still another embodiment ofthis invention can be connected;

FIG. 32 describes a data processing apparatus according to still anotherembodiment of this invention;

FIG. 33 is a block diagram of a control system in the embodiment in FIG.32;

FIG. 34 is a flowchart describing the control system in the embodimentin FIG. 32;

FIG. 35 shows a head-mounted image display device according to stillanother embodiment of this invention;

FIGS. 36 and 37 are a front and side views of an optical systemrespectively in the embodiment in FIG. 35;

FIGS. 38 and 39 show a display method used when the head-mounted imagedisplay device is used to edit still pictures. FIG. 38 is a modeldrawing showing the display method of this invention in which an imageand its attributes are displayed in a first and second display areasthat are located on the right and left of the screen, respectively;

FIG. 40 is a flowchart showing a process executed in the embodiment inFIG. 35;

FIG. 41 is a flowchart describing a voice input procedure in theembodiment in FIG. 35;

FIG. 42 is a model drawing showing an embodiment of the display methodof this invention wherein the head-mounted image display deviceaccording to this invention is used to simulate motion pictures andwherein an image and its attributes are displayed in a first and seconddisplay areas that are located on the right and left of the screen,respectively. FIG. 42(a) shows the results of a simulated image. FIG.42(b) shows the trace of the airplane.

FIG. 42(c) shows the speed of the airplane. FIG. 42(d) shows thehierarchical segmentation of the image segments;

FIG. 43 is a model drawing showing a data processing apparatus accordingto still another embodiment of this invention;

FIG. 44 is a block diagram showing a control system in the dataprocessing apparatus in FIG. 43;

FIG. 45 is a model drawing describing a data processing apparatusaccording to still another embodiment of this invention;

FIG. 46 is a model drawing describing a data processing apparatusaccording to still another embodiment of this invention; and

FIG. 47 is a block diagram showing a control system in the dataprocessing apparatus in FIG. 46.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiment 1

FIG. 1 shows an embodiment in which an operator 1 is sitting on a chair8 to perform operation on a desk 7. The operator 1 has a head-mountedimage display device 2 (hereafter referred to as a “display device”)mounted on his or her head. A computer 3 is disposed under the desk 7 totransmit image display information to the display device 2 via aconnection cord (see FIG. 2). Furthermore, the computer 3 has connectedthereto a foot switch 4 that acts as a trigger input means.

As shown in FIG. 2, the display device 2 has a spectacles-like frame 101in which a drive circuit 105 is housed, and one end of the connectioncord 106 is connected to the drive circuit 105. The drive circuit 105and the computer 3 constitute an image generation means according tothis invention. The frame 101 also has mounted in front thereof a liquidcrystal panel 102 comprising a lateral pair of liquid crystal displaydevices and a back light 104 that illuminates the liquid crystal panel102 rearways, and an enlarging lens 103 is disposed on the inner surfaceof the frame 101 between the liquid crystal panel 102 and the operator'seyes 120. Image display information sent from the computer 3 via theconnection cord 106 is decoded by the drive circuit 105 and displayed onthe liquid crystal panel 102.

The liquid crystal panel 102 is of a very small full color or monochrometype with a screen size of about 0.7 to 1.3 inch, and an image isenlarged by the enlarging lens 103 to about 14 inch at a visual range of0.5 to 1.0 m and viewed by the operator 1 as a virtual image. The liquidcrystal panel 102, the back light 104, and the enlarging lens 103 areeach provided for each of the operator's eyes 120, and ordinarilyarranged so as to display the same screen to allow the eyes to see asingle display screen.

The display screen is located so as to be seen by the operator when heor she looks to a direction somewhat above the immediate front shown byarrow A in FIG. 2, and a lower part 122 has a high transmittance withinthe range of about 70 to about 95% so that when looking somewhatdownward as shown by arrow B, the operator can look near his or herhands through the lower part 122 of the front surface of the frame 101.The upper part 123 of the front surface of the frame 101 has a lowertransmittance within the range of 0 to about 10% so as to preventextraneous light from entering the frame.

Furthermore, the frame 101 has mounted thereon an angle sensor 107 thatdetects the direction and angle of the operator's head 1 when it movesvertically or horizontally and that transmits detected data to thecomputer 3 via the connection cord 106. An infrared ray LED 108 ismounted over the enlarging lens 103, and a photosensor 109 is mountedunder the lens 103 to detect the direction of the operator's line ofsight and winking.

Referring FIG. 1 again, when looking somewhat upward, the operator 1 canview a display screen on the liquid crystal panel 102 as a virtualsubscreen 5 on a virtual display screen 6 at a distance about 0.5 m to1.0 m apart from the operator, and when looking somewhat downward, he orshe can see an operation on the desk 7.

FIG. 3 shows a scene as seen by the operator 1. In this figure, both theentire virtual subscreen 5 b and the lower part of a virtual subscreen 5c which are contained within the range shown by a display frame 51corresponding to the maximum display range of the liquid crystal panel102 are shown somewhat above the field of view. The desk 7 at which theoperator 1 is performing an operation and the operator's hands 131 areshown somewhat below the field of view. When the operator 1 turns his orher head to the right, the display device 2 also moves to the right, andthe display frame 51 accordingly moves to the right. The angle sensor107 detects the direction and angle of the operator's head, and thevirtual subscreen 5 from the computer 3 scrolls leftward according tothe detected angle. The virtual subscreen within the display frame 51changes to the subscreen 5 c, then changes to the subscreen 5 d, . . . ,and so on. This is also true in the vertical direction. In this case, ifthe output of the angle sensor 107 and the distance over which thevirtual subscreen 5 scrolls have been adjusted in advance, a fixedrelative locational relationship can be maintained between the desk 7and the virtual subscreen 5. By moving his or her head, the operator 1can view part of a large number of virtual subscreens 5; it appears tothe operator as if the subscreens were stuck to the virtual displayscreen 6 fixed to the space.

The drive circuit 105 in the display device 2 comprises a CPU 132, animage storage device 133, and an image memory 134, as shown in FIG. 4.The image storage device 133 has image data from the computer 3 storedtherein, and the image memory 134 stores image data required to displaya single screen (corresponding to the display frame 51) on the liquidcrystal panel 102. The image storage device 133 (and the CPU 132 ) maybe built into the computer 3.

The flowchart in FIG. 5 shows the operation of a control system in FIG.4. The image storage device 133 regularly loads image data from thecomputer 3, and stores, for example, image data for the virtualsubscreens 5 a to 5 e shown in FIG. 3. The CPU 132 checks whether or notthe output of the angle sensor 107 has changed, and if it changes whenthe operator 1 moves his or her head, computes an address required todisplay a screen within the display frame 51 according to the change inangle to output it to the image storage device 133. The image storagedevice 133 cuts image data corresponding to the address to output it tothe image memory 134. The liquid crystal panel 102 displays the imagedata. In this manner, the operator 1 can view a large number of virtualsubscreens 5 simply by moving his or her head.

Next, a method for moving any one of the large number of virtualsubscreens 5 on the virtual display screen 6 is described.

First, a method for specifying any one of the large number of virtualsubscreens 5 is described on the basis of the flowchart in FIG. 6.

The position of the display frame 51 is determined by the output of theangle sensor 107 as described above. That is, this output determines theaddress in the image memory 134 which corresponds to the virtual displayscreen 6. In addition, the photosensor 109 can detect what portionwithin the display frame 51 the operator is viewing. Thus, thephotosensor enables the identification of the portion on the virtualdisplay screen 6 which the operator is viewing, that is, the portionthat corresponds to a particular address in the image memory 134. Whenthe operator stamps on the foot switch 4 that acts as a trigger inputmeans, the identified address is confirmed to specify the virtualsubscreen 5 including the address, resulting the movement of thesubscreen (this is like the movement of a window on the Windows screenusing a mouse. The movement of the operator's line of sight correspondsto the movement of the mouse whereas the foot switch corresponds to theclick button of the mouse). Thus, when stamping on the foot switch 4while viewing an arbitrary virtual subscreen 5 to be moved, the operatorcan specify the portion within the virtual subscreen 5 which theoperator 1 is viewing.

Next, a method for moving the specified virtual subscreen 5 is describedon the basis of the flowchart in FIG. 7.

The virtual subscreen 5 specified in the above manner moves on the basisof the moving direction and distance determined from the most recentoutput of the photosensor 109 and the angle sensor 107. That is, thedirection of the operator's line of sight is determined from the outputof the photosensor 109 to determine where to move the virtual subscreen5 within the display frame 51, and the direction of the operator's headis detected from the output of the angle sensor 107 to determine whereto move the virtual subscreen 5 on the virtual screen 6. The virtualsubscreen 5 is thus moved, and the destination is updated until atrigger signal is input. When the operator stamps on the foot switch 4,the specified virtual subscreen 5 stops moving and is then fixed to theoperator's observation point. In this manner, an arbitrary virtualsubscreen 5 can be moved freely to a desired position on the virtualscreen 6.

It is then assumed that the operator 1 is experimenting on an electroniccircuit on the desk 7.

Specifications and other materials for parts to be used in theexperiment are registered in the computer 3 in advance to display themas a plurality of virtual subscreens 5 on the virtual screen 6. Theoperator 1 can view any material by displaying a desired virtualsubscreen 5 within the display frame 51. Furthermore, stamping on thefoot switch 4 while viewing a desired virtual subscreen 5 causes thatvirtual subscreen 5 to be specified, moving the operator's head enablesit to be moved to a position in which it can be seen most clearly, andstamping on the foot switch 4 again enables it to be fixed on thevirtual screen 6.

If the display is complicated due to a large number of virtualsubscreens 5 on the virtual screen 6, after specifying the virtualsubscreen 5 as described in FIG. 6, the combination of the number oftimes of stamps on the foot switch 4 and the interval between the stampscan be adjusted so as to send to the computer 3 an instruction foriconizing specified virtual subscreens 5 to reduce their sizes, therebyobtaining a clear display.

If displayed virtual subscreens 5 are too small to see them clearly, thecombination of the number of times of stamps on the foot switch 4 andthe interval between the stamps can be adjusted so as to send to thecomputer 3 an instruction for enlarging specified virtual subscreens 5.

As described above, this embodiment enables a large number of materialsto be referenced without putting the surroundings in disorder, therebyavoiding the decrease in working efficiency. In addition, sinceoperation can be achieved simply by moving the operator's head, eyes,and foot, and his or her hands are not occupied in the operation,accidents such as falling of a part being experimented on or damagethereto can be avoided even during a complicated experiment.Furthermore, since required materials can be placed in positions inwhich they can be seen most clearly, the movement of eyes between a partbeing experimented on and the relevant material can be minimized toreduce the operator's fatigue.

Even if a large number of virtual subscreens are displayed, they can bedistributed over the virtual screen and are enabled to scroll bymovement of the operator's head, thereby preventing them fromoverlapping each other on the screen.

Furthermore, by connecting instrument used for the experiment to thecomputer 3 in advance, the results of measurement can be displayed closeto the operator as a virtual subscreen 5 to enable the experiment to beconducted more efficiently.

Embodiment 2

FIG. 8 shows an embodiment in which the operator 1 is repairing anapparatus 22. The display device 2 is mounted on the operator's head,and an angle sensor 21 is mounted on the operator's chest, and aportable computer 13 and a push switch that acts as a trigger inputmeans are mounted on the operator's waist. The display device 2 is thesame as in Embodiment 1, and its description is thus omitted. Thedisplay device 2 is connected to the computer 13 through a connectioncord.

In looking somewhat upward, the operator 1 can view the display screenof a liquid crystal panel 102 as a virtual subscreen 5 on a virtualdisplay screen 6, while in looking somewhat downward, the operator canview the apparatus 22 being repaired.

FIG. 9 shows a scene as seen by the operator 1. In this figure, both theentire virtual subscreen 5 b and the lower part of a virtual subscreen 5c which are contained within the range shown by a display frame 51corresponding to the maximum display range of the liquid crystal panel102 are shown somewhat above the field of view. The apparatus beingrepaired is shown somewhat below the field of view. When the operator 1turns his or her head to the right, the display device 2 moves to theright, and the display frame 51 accordingly moves to the right. Thevirtual subscreen 5 from the computer 13 then scrolls leftward accordingto the angle detected by an angle sensor 107 provided in the displaydevice 2. The virtual subscreen within the display frame 51 changes tothe subscreen 5 c, then changes to the subscreen 5 d, . . . , and so on.This is also true in the vertical direction. In this case, if the outputof the angle sensor 107 and the distance over which the virtualsubscreen 5 scrolls have been adjusted in advance, a fixed relativelocational relationship can be maintained between the surrounding spaceand the virtual subscreen 5. It appears to the operator as if the largenumber of subscreens 5 were stuck to the virtual display screen 6 fixedto the space.

If the operator 1 stoops or moves to repair another part of theapparatus 22, the angle sensor 21 mounted on the operator's chestdetects the operator's movement to inform the computer 13 of it. Thecomputer 13 arithmetically processes the output of the angle sensor 21and the output of the angle sensor 107 provided in the display device 2,and if it determines that the output of the angle sensor 21 isequivalent to the output of the angle sensor 107, treats this as themovement of the entire body and allows the display on the virtual screen6 to remain unchanged. That is, the relative locational relationshipbetween the operator 1 and the virtual screen 6 is maintained constant,and the virtual screen 6 follows the movement of the operator's body. Inthis manner, the virtual screen 6 is always displayed around theoperator 1, and the virtual subscreen 5 can be viewed as required simplyby looking somewhat upward, whatever posture the operator is assuming.

Furthermore, pressing the push switch 14 while viewing a virtualsubscreen 5, as described in Embodiment 1, enables that subscreen 5 tobe specified on the basis of the direction of the operator's line ofsight detected by a photosensor 109 and the output of the angle sensor107. Then, by moving the specified virtual subscreen 5 based on theoutput of both the photosensor 109 and the angle sensor 107 andsubsequently pressing the push switch 14 again, the specified virtualsubscreen 5 can be fixed to the current observation point. In thismanner, an arbitrary virtual subscreen 5 can be moved to a desiredposition on the virtual screen 6.

Next, it is assumed that the operator 1 is actually repairing theapparatus 22. Instruction manuals and specifications required forrepairs are registered in the computer 13 in advance. A plurality ofvirtual subscreens 5 are displayed on the virtual screen 6, that is,stored in an image storage device 133. A desired instruction manual canbe viewed by moving the operator's head to display the correspondingvirtual subscreen 5 within the display frame 51. Furthermore, bypressing the push switch 14 while viewing a desired virtual subscreen 5,moving the operator's head and line of sight to a position in which thesubscreen can be seen clearly, and then pressing the push switch 14again, the virtual subscreen can be fixed to the virtual screen 6. Inaddition, even if the operator moves to repair another part of the sameapparatus, the virtual screen 6 remains in the same position around theoperator 1, so required instruction manuals can be referenced whateverworking posture the operator is assuming.

As an alternative to the trigger means described in Embodiments 1 and 2,the output of photosensor 109 is obtained, for example, when theoperator gives two consecutive quick winks. This enables both hands tobe used for repairs even in Embodiment 2. It is clear that voices orbreathing can also be used as a trigger means.

Embodiment 3

The display device 2 in the embodiment shown in FIG. 10 has aspectacles-like frame 101 including the liquid crystal panel 102, theenlarging lens 103, the back light 104, and a half mirror 135. Theoperator 1 can view an enlarged display image on the liquid crystalpanel 102 and an external image (the keyboard) through optical paths 136and 138, respectively. These images can be synthesized by the halfmirror 135 and viewed by the operator 1. The operator 1 thus feels as ifthe liquid crystal display image were present outside the frame (nearthe keyboard), resulting in improved workability.

With this configuration, however, a few problems may occur. First, theliquid crystal display image and the external image simply overlap eachother, so the display image cannot be distinguished clearly. Second,since the distance between the operator and the virtual image cannot bematched easily with the distance between the operator and the keyboard,focusing for eyes must be carried out each time the field of view isswitched between the liquid crystal display image and the keyboardcontained in the synthesized image, resulting in a marked fatigue of theoperator.

These problems can be solved by employing part of an outer sheath casefor the data processing apparatus and the keyboard as a screen. That is,the rear face 141 of a cover for the keyboard 140 of the computer 3 isused as a screen. This screen consists of an opaque material with asingle color such as white, black, or blue, and is effective inimproving the visibility of a liquid crystal display image that isallowed to overlap the screen image. The color and reflectivity of thescreen are depends on the degree of this effect.

As in the example shown in FIG. 12, the operator 1 can longitudinallymove relative to the operator 1, the position in which the virtualsubscreen 5 for a liquid crystal display image is formed by moving theliquid crystal panel 102 in the direction 139 (along the optical path136 ) in FIG. 10. Then, matching this position with the screen 141enables the above problems to be solved.

Since the external image that overlaps the liquid crystal display imageis the monochrome screen, it does not substantially disturb thevisibility of the display image. That is, the display image has a highvisibility, leading to improved workability. In addition, the virtualsubscreen 5 for a liquid crystal display image is formed on the screen141 close to the keyboard 140, so focusing for eyes need not be executedwhen the field of view is switched between the display image (thevirtual subscreen 5) and the keyboard 140, resulting in a reducedfatigue of the operator.

Consequently, the above embodiment enables the implementation ofportable computers and word processors that have high portability andvisibility and that do not affect the operator's health.

The characteristics of the screen is not limited as long as they meetthe objectives of this invention. For example, attaching a polarizingplate to the screen can reduce the brightness of the external image withthe brightness of the liquid crystal display image maintained, therebyenabling high contrast display to be realized.

Embodiment 4

The display device in FIGS. 13 and 14 is of a single eye type. In thesefigures, the cable for connecting the drive circuit to the main body isomitted for simplification.

An image displayed on the liquid crystal panel 102 is enlarged by theenlarging mirror 145, and the operator's right eye sees a large displayimage in front of the operator, while his or her left eye sees theexterior through the transparent spectacles-like frame 101. The problemin single eye-type display devices (the visibility of the display imageis bad because the external image overlaps the display image, and thedifference between focusing for the right and left eyes causes asignificant fatigue of the operator) can be solved by adjusting thedistance between the liquid crystal panel 102 and the enlarging mirror145 so as to form on the screen a virtual subscreen 5 that is a displayimage so that the operator's left eye can see the screen 141. That is,since the virtual screen on the liquid crystal display screen is formedon the monochrome and uniform screen 141, the visibility of the liquidcrystal display screen is improved, and an equal focusing for the rightand left eyes serves to reduce the operator's fatigue to almost the samelevel as in ordinary displays.

Consequently, the above embodiment can provide portable computers andword processors that have a good portability and visibility and that donot affect the operator's health.

Embodiment 5

In the embodiment in FIG. 15, a signal from the input device 150 isinput to the computer (an arithmetic unit) 3 via the connection cord151, and image display information output from the computer 3 istransmitted to the head-mounted display device 2 via the connection cord106. The head-mounted display device 2 enables the simultaneous displayof that input information such as the conditions of the input device 150such as a keyboard or a data glove which is output by the computer 3 andthe results of arithmetic operations from the computer 3, and indicateswhat keys can be currently used for input and what key the operator istouching.

The display screen of the head-mounted display device 2 is long, and iscomposed of an arithmetic operation result display screen 152 fordisplaying the results of arithmetic operations within the virtualsubscreen 5 and a virtual keyboard display screen 153 for displaying theconditions of the input device. By using as the input device 150, forexample, a data glove having a means for measuring the transmittance oflight through optical fibers to detect the bending of the operator'sfingers and using a magnetic sensor to detect the relative position ofthe operator's hands, or a keyboard with a sensor having a positiondetection means such as a resistant film-like tablet, the input positioncan be detected, and the results of arithmetic operations can be viewedwhile a key assignment means 156 and an image synthesizing means 159 inthe computer 3 are used to check on the virtual keyboard display screen7 what key is being pressed.

Next, a method for outputting the results of arithmetic operations onthe arithmetic operation result display screen 152 and displaying theconditions of the input device 150 on the virtual keyboard displayscreen 153 is described with reference to the block diagram in FIG. 17.The input device 150 detects positional information 154 for thepositions of the operator's hands and fingers and on/off information 155indicating whether or not any finger tip is pressing a key to transmitthis information to the computer 3. The computer 3 uses the keyassignment means 156 to assign keys over the input range of the inputdevice 150 for each application in advance in order to determine whatkeys correspond to input positional information 154, and determines fromthe on/off information 155 whether or not the key is activated in orderto transmit the results to an arithmetic operation section 158 and animage synthesizing means 159 as input information 157. The arithmeticoperation section 158 performs arithmetic operations based on the inputinformation 157 to transmit the results 160 to the image synthesizingmeans 159. The image synthesizing means 159 continuously outputs theconditions of the input device 150 such that which key the operator'shand is touching based on the input information and the on/off state ofthe key, and outputs to the head-mounted display device 2 a signal as animage signal 161 which synthesizes the results of arithmetic operation160 from the arithmetic operation section 158 into one image each timesuch an operation is carried out. The input device 150 simply transmitsthe positional information 154 and the corresponding on/off information155 to the computer 3, thereby enabling the key assignment means 156 toassign various keys without depending on the number of keys on the inputdevice 150 and also allowing the virtual keyboard display section 153 todisplay various interfaces.

The flow of signals in the control system in FIG. 17 is described withreference to FIG. 18. The input device 150 detects the input positionand the corresponding on/off information (S10). The key assignment means156 determines to what key the input position corresponds, and transmitsthe contents of the corresponding key to the arithmetic operationsection 158 (S11). The key assignment means 156 then determines whetheror not the key is being pressed, and if so, informs the arithmeticoperation section 158 of this state (S12). The image synthesizing means159 then generates an image on the virtual keyboard display screen 153based on the input position and the on/off information (S13). Thearithmetic operation section 158 performs arithmetic operations based ontransmitted information to generate an arithmetic operation resultdisplay screen 152 (S14). The image synthesizing means 159 synthesizesas a single screen the virtual keyboard display screen 153 and thearithmetic operation result display screen 152 generated by thearithmetic operation section 11 to generate a virtual subscreen (S15).

The input device 150 includes resistant film-type devices, a collectionof small switching elements, and various other devices. Next, theprocess executed by the assignment means 156 when a resistant film-typeinput device is used for key and pointing input is described in detail.

The assignment means 156 first determines the input method of the inputdevice 150 (S20). If the input device 150 uses a key input method, theassignment means 156 divides the input plane into key areas (S21). Theassignment means subsequently determines whether or not the input methodhas been switched (S22), and if not, receives the input signal (X, Y)(S23). It then examines to which key area the input signal (X, Y)corresponds (S24), and outputs the code for the corresponding key (S25).

If the assignment means determines in S20 that the input device uses apointing method, it then determines whether or not the input method hasbeen switched (S26). If not, it receives the input signal and stores itin a buffer (X₀, Y₀) (S27). It then receives the next input signal (X₁,Y₁) to determine the difference between the this signal and thepreceding signal (X₀, Y₀) (S28). The assignment means determines whetheror not the difference is, for example, 2 cm or longer (S29), and if so,treats this input as the result of clicking on (X₀, Y₀) (S30). If thedifference is shorter than 2 cm, the first signal is moved by thecorresponding amount of movement. The above processing between step S26and step S31 is performed in 0.01 second or shorter.

A shown in FIG. 20, in the display device 2 in FIG. 15, one end of theconnection cord 106 is connected to the spectacle-like frame 101 formounting the display device 2 on the operator's head in order to supplyimage signals and power to the drive circuit 105 and back light 104disposed in the upper front of the spectacle-like frame 101. An imagesignal input to the drive circuit 105 is decoded and displayed on theliquid crystal panel 102. The displayed image information is illuminatedby the back light 104 disposed on the rear face of the liquid crystalpanel 102, and reaches the operator's eyes via a reflection mirror 170and the enlarging lens 103 located on the front surface of thespectacle-like frame 101 and in front of the respective eyes of theoperator. In another aspect, disposing a beam splitter between theliquid crystal panel 102 and the reflection mirror 135 to project animage in two different directions enables the image to reach both eyesof the user via the reflection mirror and the enlarging lens located infront of the respective eyes.

The liquid crystal panel 102 is a long and small full color ormonochrome LCD with a screen size of 1 to 2 inches, and allows a virtualsubscreen 5 about 14 inches in size to be seen by the user via theenlarging lens 103 at a distance 500 to 1,000 mm apart from the user'seyes, as shown in FIG. 20.

In the above embodiment, although the computer and the display deviceare separated from each other, the computer may be incorporated in thehead-mounted display device, and in this case, signals from the inputdevice are transmitted directly to the head-mounted display device andthe computer. This computer may be implemented with an arithmeticoperation device designed for a display device.

The interface in the above embodiment for which the key assignment meansin the arithmetic operation device executes assignment is not limited toa keyboard but may be a mouse, a joystick, or a pen.

Furthermore, the input device is not limited to a tablet or a data gloveas in the above embodiment, various input devices that can detectpositional information and on/off information may be connected to thecomputer.

As an alternative to the embodiment shown in FIG. 16, the imagesynthesizing means in the computer (the arithmetic unit) can be omittedby separately driving and displaying the arithmetic operation resultdisplay screen and the virtual keyboard display screen to therebyimplement the display section of the head-mounted display device.

Embodiment 6

The embodiment in FIGS. 21 to 29 shows the data processing apparatusaccording to this invention applied to a personal computer. FIG. 21 is aperspective view wherein the display device 2 in the above embodiment isused as the display means. A signal of image display information outputfrom the computer 3 is transmitted to the display device 2 via theconnection cable 106. To use the display means separately from themethod described below, the cable 106 is connected to the computer 3 viaa connector 23 so as to be removed easily therefrom.

The display device 2 in FIG. 21 essentially has the same constitution asthe device shown in FIG. 20 except the liquid panel 102 can be removedeasily from the frame 101. The reflection mirror 170 can be movedvertically and longitudinally and also have its angle adjusted so thatthe operator can get a clear view. The frame 101 serves to interceptextraneous light so that the image displayed on the liquid crystal panel102 can be seen clearly. The exterior, however, must be seen through theframe 101, so the frame 101 preferably has a light transmittance ofabout 70 to 80%.

In addition, in the embodiment in FIG. 21, power for driving the displaydevice 2 and image information output from the main body are transmittedto the display device 2 via a connector recess 60 provided in thecomputer 3 main body, a connector convex 23 provided on the displaydevice 2, and the connection cable 106 for connecting the computer 3main body to the display device 2.

FIG. 22 shows a screen used as the display device, and FIG. 23 shows across section of the screen.

The liquid crystal panel 102, removed from the display device 2, isplugged in a socket 24. An image displayed on the liquid crystal panel102 is illuminated by light from the back light 25, enlarged by a lens206, and projected on a screen 27. As a result, the operator can viewthe image 28. The vertical position and size of the image 28 can beadjusted by moving the liquid crystal panel 102 vertically orlongitudinally in the socket 24. The screen is not necessarily a whitepanel, but may be a flat surface without patterns such as a wall.

FIGS. 24 and 25 show the correlationship between the vertical positionof the liquid crystal panel and the vertical position of the image 28.These figures show that moving the liquid crystal panel 102 verticallyenables the vertical position of the image 28 to be adjusted.

FIGS. 26 and 27 show the correlationship between the longitudinalposition of the liquid crystal panel and the size of the image 28. Thesefigures show that moving the liquid crystal panel 102 longitudinallyenables the size of the image 28 to be adjusted. When the distancebetween the liquid crystal panel 102 and the enlarging lens 26 and thedistance between the enlarging lens 26 and the screen 27 are referred toas S₁ and S₂, respectively, expression (1) is given. (f) is a focallength. In addition, when the size of the image is referred to as β,expression (2) is given.[Expression1] $\frac{1}{S_{2}} = {\frac{1}{S_{1}} + \frac{1}{f_{1}}}$[Expression2] $\beta = \frac{S_{2}}{S_{1}}$

In expression (1), S₂ increases with decreasing S₁, while S₂ decreaseswith increasing S₁. Consequently, the size of the image β increase withdecreasing S₁ while it decreases with increasing S_(1.)

FIG. 28 shows a perspective view of the structure of the socket 24, andFIG. 29 shows an exploded perspective view of the socket 24.

The liquid crystal panel 102 has its vertical and longitudinaldirections adjusted by a pin 54 and an eccentric pin 55, respectively.The liquid crystal panel 102 is inserted in the direction A in FIG. 29.The socket 41 fixes the liquid crystal panel 102. The pin 54 is insertedinto a tapped hole 51 via a hole 42 to fix the socket 41 to a socket 44.Thus, the rotational movement of the pin 54 causes the socket 41 to movevertically and thus the liquid crystal panel 102 to move vertically.Pieces 46, 47 of the socket 44 engage the socket 41, and the pieces andthe socket 44 are fixed longitudinally together. Rotating the eccentricpin 55 in a long hole 50 enables longitudinal movement. Pins 52, 53 areinserted into long holes 48, 49 to prevent lateral deflection. Windows43, 45 allow light from a light source to pass through.

Although the above embodiments have shown an example of a liquid crystalpanel of a detachable structure, the liquid crystal panel in the displaydevice 2 and the liquid crystal panel in the computer main body can notnecessarily be removed from the respective devices, and the liquidcrystal panels may be fixed and a display switch is provided to switchthe display means for the computer.

In addition, although the above embodiment has referred to the use of apersonal computer, this invention is not limited to the aboveembodiments and is applicable to data processing apparatuses such asword processors. The single eye-type head-mounted display device hasbeen described, but this invention is applicable to double eye-typehead-mounted display devices. Furthermore, although the aboveembodiments have shown the use of a spectacle-type head-mounted imagedisplay device, this invention is applicable to other types ofhead-mounted image display devices such as goggle and helmet types.

If the data processing apparatus of the above configuration is usedtogether with a head-mounted image display device as a display means,only the operator can directly view the contents of display, so theoperator can process secret information without caring about peoplearound the operator. The portability of this data processing apparatusis also good because the operator can use the apparatus in a naturalposition in a limited space as in an airplane or a train. In addition,the field of view can be moved easily between the virtual image viewedthrough the display device and the exterior including the keyboard,leading to the improved operability of the data processing apparatus.

If a plurality of people desire to simultaneously see image displayinformation output from the data processing apparatus, the liquidcrystal panel in the apparatus main body may be used to display imagedisplay information on the screen or the like. Such switching of thedisplay means can be carried out easily.

Embodiment 7

FIG. 30 shows a conventional computer in which its display section canbe removed from its main body. A computer main body 3 a is engaged witha display 63 via a connector recess 61 and a connector convex 62 a.

FIG. 31 shows an embodiment in which the display device 2 in the aboveembodiment is mounted on the computer 3 in FIG. 30. In this embodiment,the computer main body. 3 a is connected to the display device 2 via theconnector recess 61, the connector convex 62, and the connector cable106.

If both the connector convex 62 a of the display 63 in FIG. 30 and theconnector convex 62 of the cable 106 in FIG. 31 conform to the samestandard, both the conventional display 63 and the display device 2according to this invention can be connected to the same main body. Thatis, a data processing apparatus is realized wherein the conventionaldisplay 63 and the display device 2 according to this embodiment can beexchanged easily.

Although the above embodiments use a computer as the data processingapparatus, this invention is not limited to these embodiments, and isapplicable to other data processing apparatuses such as word processors.The single eye-type head-mounted display device has been described, butthis invention is applicable to double eye-type head-mounted displaydevices. Furthermore, although the above embodiments have shown the useof a spectacles-type head-mounted image display device, this inventionis applicable to other types of head-mounted image display devices suchas goggle and helmet types.

As is apparent from the above explanation, this embodiment isadvantageous in that the display device can be removed easily from thedata processing apparatus. In addition, the use of connectors conformingto a common specification enables exchanges between a conventionaldisplay and the display device according to this embodiment.Furthermore, the back light in the display device of this embodimentconsumes less power than displays used in conventional portable dataprocessing apparatuses. As a result, the life of the battery can beextended, resulting in the significantly improved portability of thedata processing apparatus according to this embodiment.

Embodiment 8

In the image display system in FIG. 32, output results from the computer3 are displayed on the display 63 and the display device 2 mounted onthe operator's head. The user 1 can execute processing while slightlymoving his or her glance to view the image displayed on the display 63and the virtual subscreen 5 created by the display device 2 mounted onhis or her head. The virtual subscreen 5 is generated by allowing animage displayed on the liquid crystal panel 102 to reach the user's eyesvia an enlarging reflection mirror section 64.

In the control system of the display device in FIG. 33, an output imagefrom the arithmetic unit 70 is assigned by an output informationassignment means 72 to the respective image output devices for output,and then written to a graphic RAM 73 in an image output control section71. The information written to the graphic RAM 73 is managed on thebasis of addresses, information output from the arithmetic unit 70 isoutput to the display device 2 and the display 63. For example, the sameinformation may be displayed on both the display device 2 and thedisplay 63, or separate data sets may be displayed on the display device2 and the display 63, respectively, and the information can also betransferred to an arbitrary image display device.

The flowchart in FIG. 34 shows a process of outputting to a plurality ofimage display devices image information output from the arithmetic unit.As if Image display devices of apparently large capacities wereconnected to the arithmetic unit, output information from the arithmeticunit is read into these display devices (S51). It is determined whetherthe read information comprises an image or controls display areas (S52).If the information is to control the display area, a memory address areathat must be set in the graphic RAM 73 to display an image on eachdisplay device is determined on the basis of this information (S53). Ifthe information comprises an image, parameters such as a display area(addresses) are determined on the basis of the memory address area(S54). It is then determined whether or not the image information shouldbe contained in the display area determined in step S54 (S55), that partof the information which is not associated with the display area isignored during processing (S56) and the rest of the information whichshould be contained in the display area is address-converted (S57) andwritten to the graphic RAM (S58). Each of the image display devicedisplays the information in the assigned address area as required.

The user views the virtual subscreen 5 in part of his or her field ofview and the other image display devices in the rest of his or her fieldof view, and can see the plurality of image display devices simply byslightly moving his or her line of sight.

Although, in the above embodiment, two image display devices areconnected to the portable computer, the numbers of arithmetic units andimage display devices are not limited, and in a system that processes alarge amount of information using a plurality of arithmetic units, theresults of processing can be displayed on the plurality of image displaydevices with only the information to be monitored displayed on ahead-mounted display device.

Embodiment 9

In the display device in FIG. 35, a spectacles-like frame 101 hasmounted therein a liquid crystal panel display section 172 for the lefteye and a liquid crystal panel display section 173 for the right eye.These liquid crystal panel display sections 172, 173 form a virtualimage on the retina of the eyes via a convex lens 103 and a mirror 170in proportion to the strength of each image by irradiating the rear ofthe liquid crystal panel 102 from behind with extraneous light or lightfrom the back light 104. A drive circuit for controlling these liquidcrystal panels 102 may be configured integrally with or separately fromthe spectacles-like frame so as to form images on the liquid crystalpanel.

A microphone 174 is joined with the frame with a flexible joint 175 thatis fixed to the position of the user's ears and which can be bent towardthe user's mouth. A tracking sensor 176 that acts as a positiondetection means is installed in the center of a frame 177 disposed so asto link the user's right and left ears together. The tracking sensor 176detects the direction of the operator's line of sight using the momentof inertia effected when the user's head is turned. The display devicein this embodiment includes a mouse 180 and a keyboard 181.

FIGS. 38 and 39 show an embodiment in which a still picture obtainedusing the display device shown in FIG. 35 is edited. These figuresdescribe a display method in which a first and second display areas aredisplayed laterally on the display screen.

The first display area usually contains still pictures that are shown infront of the operator. The second display area is displayed on the rightof the screen according to the amount of the user's movement detected bythe tracking sensor 176 when he or she turns to the right. The seconddisplay area displays attributes of the image data shown in the firstdisplay area. In this case, when the user turns to the right, thetracking sensor 176 detects this movement to create on the two liquidcrystal panels 102 an image such as shown in FIG. 39. This embodimentrelates to the synthesis of still pictures, in which the segments of theimage (geometric models such as a tree, a house, or a car) are detectedon the second display area and have their sizes adjusted. In this case,for example, a “car” is input from the keyboard 181 to search its designfile for an appropriate material. Searches are similarly carried out forthe house and tree, and the size of the image is also adjusted on thesecond display area. The processed images materials are transferred tothe first display area in which they are synthesized against abackground such as a mountain.

Next, the above editing process is described with reference to theflowchart in FIG. 40. The output of the tracking sensor 176 is firstinput, and the direction of the operator's line of sight is detected onthe basis of this input signal (S20). If the line of sight is determinedto be to the left, background images are input to the first display area(S21) to select among them (S22). It is then determined on the basis ofthe output of the tracking sensor 176 whether or not the operator 1 hasturned to the right (S23), and if not, the background image is thenprocessed (S24). Until the processing is finished (S25), the process ofselecting and processing a background images is repeated (S22 to S24).

When the processing has been finished (S25), it is then determinedwhether or not to add data to the image (S26). If it is determined thatdata should be added, attributes data is added to the first display area(S27).

Voice input is then used to edit the details of the layout (S28). Thisprocess is described later. When the edition is finished (S29), theseries of processing is completed.

On the other hand, if the direction of the operator's line of sight isdetermined to be to the right when the direction of the line of sight isdetected (S20), attributes data is input to the second display area(S30). A reference coordinate position is determined (S31), andattributes data is searched for (S32). When the search is finished(S33), the attributes data is processed (S34). When the processing isfinished, it is determined whether or not edition will be subsequentlycarried out (S36). If so, segments to be edited are selected (S37). Itis then determined whether or not the operator 1 is looking to the left,and if so, the reference coordinates are calculated (S39) and theprocess transfers to the above step S27. Otherwise, that is, if theoperator is determined to be still looking to the right, the processreturns to step S34 to repeat the process of processing attributes dataand selecting segments to be edited.

An audio editing process is described with reference to the flowchart inFIG. 41. Audio input does not simply consist of understanding wordsimplied by commands but employs an audio input means called“WordSpotting” for unspecified speakers in which clauses can be input.

In this audio input means, sentences are understood by neuro-computerprocessing in which nouns such as a tree, the right side, movement, theleft, and a height are registered in advance to understand the entiresentence based on the occurrence probability of each word.

In this case, a virtual image is shown within the first display area viaa wide angle lens, so the image is enlarged, and it is possible to allowthe user to feel as if he or she were in that image. The distance overwhich the image moves is set on the computer to the size of the firstdisplay area, and the distance between the point of view and an objectis calculated, in advance. During edition, a desired image can be inputin the form of a voice to edit even minor positional relations whileviewing the image in the first display area, without changing the pointof view.

The point of view can also be changed in the second display area as ifthis change were effected by camera work.

FIG. 42 shows a simulation of motion picture data using the displaydevice shown in FIG. 35. A simulated image (a) is displayed in the firstdisplay area, while its attributes are displayed in the second displayarea. The second display area shows spatial area data that varies withthe lapse of time, for example, temporally changing and graphedindications of the (b) locus of a flying airplane, (c) its speed, andthe (d) hierarchical segmentation of the image segments.

In this embodiment, the locus of the airplane can be obtained byplotting the temporally changing location of the airplane on the seconddisplay area. A speed can be set for each plotted point based on thetime vs. speed graph. If, for example, the speed is to be varied, themouse is clicked on each step point to move it upward or downward. Thesegments that appear in the image data according to a certain time basecan be cut or pasted to the image based on the graph in which the imagesegments are hierarchically expressed.

Such processing enabling the attributes of the image data to betemporally varied is very effective in editing segmented data duringpostproduction. The efficiency of edition is further improved byapplying the depth of each image material and a parallax thatcorresponds to the difference between the right and left eyes in orderto achieve three-dimensional display in which the operator feels as ifhe or she were viewing actual scenes as well as hierarchical displaythat expresses the depth well.

Embodiment 10

In the data processing apparatus in FIG. 43, an arithmetic unit 81 is aportable type such as conventional notebook personal computers orpersonal computers. This arithmetic unit 81, however, does not include aconventional open-type display but comprises an arithmetic unit mainbody 82 consisting of an arithmetic section, a memory section, and akeypad; and a cover 83 for protecting the upper surface of the keypadduring storage or transportation. As in conventional data processingapparatuses, the cover 83 is connected to the arithmetic unit main body82 via a hinge-like connection 84 so as to be openably and closablyretained. The inner surface of the cover 83 has a buffering structureformed of an elastic body, and part of it forms a housing section 86 forhousing the display device 2. The display device 2 comprises aspectacles-like frame 101 in which the above optical system and drivecircuit are housed, and is mounted on the user's head so as to cover hisor her face. A bow 88 is formed at both ends of the frame 101 so as toextend to the user's ears with hinges 89, and can be folded and housed.A communication cable 106 from the drive circuit protrudes from one endof the bow 88 and connects to the arithmetic unit main body 82 throughone end of the housing section 86.

In operation, when the head-mounted image display device 2 is fixed tothe user's head via the frame 101, and the arithmetic unit 81 is driven,then the drive circuit and the optical system in the display device 2enlarge an image information output from the arithmetic unit 81 anddisplay it as a virtual image at a definite distance from the operator'sface. In storage or transportation, the bow 88 on either side of thedisplay device 2 is folded and the display device is housed in thehousing section 86. The communication cable 106 is also folded andhoused in the housing section 86, and the cover 83 is closed. Thisapparatus then constitutes an integral structure like conventionalportable data processing apparatuses. Consequently, it is possible toimprove the portability and operability of the data processing apparatuswhile maintaining conventional advantages such as the ability to prepareconfidential documents and to avoid locational limitations such as theneed to fully open the display.

The operation of the above embodiment is explained with reference to theblock diagram in FIG. 44. An electric current supplied from an externalpower supply AC input is allowed to branch into two flows at a powercircuit 96 in the arithmetic unit 81, one of which is fed to thearithmetic operation section 97. The other flow and the output of thearithmetic operation section 97 are supplied to the drive circuit 105 inthe display device 2 via the connection cable 106, and then convertedinto a liquid crystal driving signal to drive the liquid crystal panel102. In this block diagram, components that are not required for theexplanation are omitted and are not described. In addition, the size andweight of the display device 2 can be further reduced by dividing thedrive circuit 105, which is housed in the display device 2 in thisembodiment, into two parts, one of which is incorporated in thearithmetic unit 81 in such a way that this configuration will not resultin the occurrence of noise.

Embodiment 11

The data processing apparatus in FIG. 45 has the same basicconfiguration as in the preceding embodiment. In this embodiment, ahousing section 90 for housing the display device 2 is formed inside thearithmetic unit main body 82, and the connection cable 106 is connectedto both the power supply in the arithmetic unit 81 and the arithmeticoperation section. In this constitution, although the volume of thearithmetic unit main body 82 main body somewhat increases, the need of alarge cover is eliminated to reduce the size and weight of the apparatusand thereby improve portability, and the need of an open-type cover iseliminated to reduce the locational limitations, thereby furtherimproving usability.

Embodiment 12

The data processing apparatus in FIGS. 46 and 47 also has the same basicconfiguration as in the two preceding embodiments. Like components carrylike reference numerals, and their description is thus omitted.

In this embodiment, the arithmetic unit 81 has a charger therein, andthe display device 2 has a chargeable battery therein. In addition, thearithmetic operation section 97 and the drive circuit 105 communicatewith each other by radio. The use, storage, or transportation of thisapparatus thus requires a smaller space. The arithmetic unit 81 has acharger 98 to which electric currents are supplied from a power supply96 and a transmission circuit 99 to which signals are delivered from thearithmetic operation section 97. A signal supplied to the transmissioncircuit 99 is modulated in the circuit 99, then transmitted and receivedvia antennas 201, 202, and demodulated in a receive circuit 203. Thesignal is finally input to the drive circuit 105. The display device 2has a chargeable battery 204 therein for delivering drive currents tothe drive circuit 105.

An electrode contact 205 connected to the charger 98 is exposed on theend face of the housing section 90. If, for example, the head-mountedimage display device 2 is to be housed in the housing section 90 byinserting the antenna 202 side into the section 90, an electrode contact206 is provided at the antenna 202 side end of the frame 101 thatengages the electrode contact 205 when the display device is housed inthe main body, and connected to the battery 204. In this configuration,when the head-mounted image display device 2 is housed in the main body,the electrode contact 205 of the charger 98 and the electrode contact206 of the battery 204 are connected together. The battery 204 is thuscharged while the display device 2 is housed in the main body, that is,while the device 2 is out of use.

In this constitution, the arithmetic unit 81 and the display device 2are completely separated from each other, thereby substantiallysimplifying the use of the data processing apparatus 81 as well as themounting and removal of the display device 2 from the housing section90. In addition, even when the display device 2 is housed in the samemanner as in the embodiment in FIG. 43, similar effects can be producedby adding the above components to the apparatus.

The display device in each of the above embodiments need not necessarilybe of a single eye type in which an optical system is disposed over onlyone eye while the other eye views an image, but may be of double eyetype in which an optical system is disposed over each eye. In this case,the optical axis of each optical system is preferably directed inward sothat the glances of the two eyes intersect each other at an imagevisible distance (a distance at which a virtual image can be perceived),as in the natural way of looking at objects. In addition, the abovecomponents are not limited to the above contents, and variousmodifications may be made thereto without departing from the spirit ofthis invention.

1. A head mount image display device wherein a liquid crystal displaypanel and enlarging optical system are housed in a frame and wherein animage generated on the liquid crystal panel is visible in a field ofview in front of said frame through the enlarging optical system, saiddevice comprising: an input device for inputting signals, with apositional sensor to detect positions of an operator's hands andfingers; a key assignment means for assigning input keys to informationregarding the positions detected by said positional sensor; anarithmetic unit for performing arithmetic operations based on datacorresponding to an input key assigned by said key assignment means; animage synthesizing means for synthesizing a virtual key board displayscreen for displaying input keys to be assigned by said key assignmentmeans, and an arithmetic operation result display screen for displayinga result of the arithmetic operation performed by said arithmetic unit,so as to display both said display screens side by side on one screen.2. An image display device according to claim 1, comprising a voiceinput means connected to said frame.